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This  article  uses  transaction  costs  to  predict  the  probability  of  incurring  a 
cost  breach  in  a  major  defense  acquisition  program  (MDAP).  As  transaction 
costs  are  not  explicitly  measured  for  MDAPs,  the  authors  use  estimates  of 
systems  engineering  and  program  management  (SE/PM)  costs  as  a  share 
of  overall  program  costs  as  a  proxy  for  transaction  costs.  Using  survival 
analysis,  a  new  approach  to  predicting  cost  breaches,  they  also  found  that 
an  increased  share  of  SE/PM  costs  in  initial  program  estimates  can  help 
predict  future  cost  breaches. 
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Controlling  cost  growth  for  a  major  defense  acquisition  program 
(MDAP)  has  been  problematic  in  the  Department  of  Defense  (DoD) 
for  many  years.  A  2007  RAND  study  of  cost  growth  in  DoD  weapon 
systems  determined  that  the  cost  of  the  46  programs  studied  was 
more  than  1.46  times  the  cost  estimate  for  Milestone  B  (program 
initiation)  (Younossi  et  al.,  2007).  According  to  the  Government 
Accountability  Office,  active  MDAPs  in  Fiscal  Year  (FY)  2011 
collectively  experienced  a  cost  growth  of  $74.4  billion  (Sullivan, 
2011). 


The  Selected  Acquisition  Report  (SAR)  was  introduced 
in  1967  to  provide  DoD  and  the  Congress  a  summary 
of  each  MDAP’s  ability  to  meet  cost,  perfor¬ 
mance,  and  schedule  objectives  agreed  upon 

I  by  the  program  manager  and  defense  acqui¬ 
sition  executive.  Program  managers  were 
now  required  to  provide  a  brief  explanation 
in  the  SAR  of  how  and  why  any  cost  breaches 
occurred. 

Based  on  evidence  that  this  was  insufficient  to 
control  cost  growth,  in  1982  Senator  Samuel 
Nunn  and  Congressman  David  McCurdy 
introduced  the  Nunn-McCurdy  Act  (1983)  to 
hold  DoD  accountable  to  Congress  for  man¬ 
agement  of  program  costs.  The  Nunn-McCurdy 
Act  became  law  with  the  FY  1983  Department 
of  Defense  Authorization  Act,  establishing  congressio¬ 
nal  oversight  for  MDAPs  that  exceed  established  cost 
thresholds.  The  Nunn-McCurdy  Act  has  been  statuto¬ 
rily  amended  a  number  of  times  over  the  years.  One  of 
the  most  significant  changes  to  the  reporting  require¬ 
ments  occurred  in  the  FY  2006  National  Defense 
Authorization  Act  (Pub.  L.  109-163),  when  Congress 
added  the  original  baseline  estimate  as  a  threshold 
against  which  to  measure  cost  growth  (National 
Defense,  2006).  The  new  standard  prevents  DoD 
from  avoiding  a  Nunn-McCurdy  breach  by 
simply  rebaselining  a  program. 
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Clearly,  the  ability  to  anticipate  cost  overruns  before  breaches  occur  would 
be  extremely  valuable  to  program  managers  and  policy  makers.  However, 
the  conventional  focus  on  MDAP  production  costs  potentially  misses  a 
critical  clue.  Angelis,  Dillard,  Franck,  and  Melese  (2008)  hypothesize  that 
the  higher  the  ratio  of  another  key  set  of  costs,  transaction  costs  (costs 
associated  with  “source  selection,...  contract  negotiation  and  management, 
performance  measuring  and  monitoring,  and  dispute  resolutions”)  relative 
to  production  costs,  the  greater  the  likelihood  of  schedule  and  cost  overruns. 

Higher  transaction  costs  are  typically  experienced  in  programs  that  involve 
greater  asset  specificity,  complexity,  and  imperfect  and  asymmetric  infor¬ 
mation— in  other  words,  programs  that  are  at  greater  risk  (Franck  &  Melese, 
2008).  Proxy  measures  first  suggested  by  Angelis  et  al.  (2008)  that  can  be 
used  to  capture  many  of  these  costs  include  systems  engineering  and  pro¬ 
gram  management  (SE/PM)  costs  regularly  reported  by  MDAP  contractors. 
It  seems  reasonable  to  assume  that  combined  SE/PM  costs  will  be  a  higher 
share  of  total  overall  estimated  costs  (production  +  transaction  costs,  as 
predicted  at  the  Milestone  B  decision  point).  Program  managers,  therefore, 
could  reasonably  anticipate  higher  costs  in  coping  with  more  complex  and 
riskier  MDAP  projects.  Unclear  is  whether  or  not  these  costs  are  explicitly 
considered  in  program  cost  estimates  or  simply  reflected  in  the  size  of  SE/ 
PM  staff  assigned  to  more  complex  or  high-risk  programs. 


Background 

Many  studies  (e.g.,  Bolten,  Leonard,  Arena,  Younossi,  &  Sollinger,  2008) 
have  examined  cost  growth  in  DoD  programs,  yet  little  research  has  been 
done  on  the  relationship  between  transaction  costs  and  cost  overruns  as 
suggested  by  Angelis  et  al.  in  2008.  A  2006  RAND  study  established  that 
MDAP  SE/PM  costs  vary  between  programs  depending  on  the  program  type 
(Stem,  Boito,  &  Younossi,  2006),  and  Angelis  et  al.  (2008)  suggested  using 
the  SE/PM  cost  as  a  proxy  for  transaction  costs  to  examine  the  relationship 
between  transaction  costs  and  cost  overruns. 

In  general,  a  program  has  two  types  of  costs:  production  costs  and  transac¬ 
tion  costs.  Production  costs  are  usually  captured  in  the  Work  Breakdown 
Structure  (WBS),  but  transaction  costs  may  not  be  adequately  captured  in 
the  WBS.  Because  traditional  cost  estimates  are  based  on  the  production 
costs  found  in  the  WBS,  they  do  not  explicitly  include  transaction  costs 
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(Angelis  et  al.,  2008).  Although  they  are  not  often  captured  in  the  account¬ 
ing  records,  the  time  and  effort  associated  with  complex  and  risky  MDAP 
transactions  represent  real  costs  to  the  organization. 

In  previous  research,  Angelis  et  al.  (2008)  examined  how  transaction  costs 
might  be  captured  in  the  cost  estimates  of  DoD  acquisition  programs. 
Angelis  et  al.  identified  a  number  of  issues  with  DoD  program  management 
cost  data  reported  for  major  weapon  systems  and  found  that  they  are  not 
well  suited  for  developing  a  cost  model  that  includes  transaction  cost  vari¬ 
ables.  As  an  alternate  approach,  they  explored  using  contractor  Program 
Management  data  from  cost  data  summary  reports  (DD  Form  1921)  and 
suggested  using  the  SE/PM  category  as  a  proxy  for  transaction  costs.  The 
DoD  (2011a)  defines  systems  engineering  as  “the  technical  and  management 
efforts  of  directing  and  controlling  a  totally  integrated  engineering  effort 
of  a  system  or  program.”  Program  management  is  defined  as  “the  business 
and  administrative  planning,  organizing,  directing,  coordinating,  con¬ 
trolling,  and  approval  actions  designated  to  accomplish  overall  program 
objectives,  which  are  not  associated  with  specific  hardware  elements  and 
are  not  included  in  Systems  Engineering”  (DoD,  2011,  p.  222).  The  ratio  of 
(SE  +  PM)  costs  relative  to  total  program  costs  offers  a  potentially  valuable 
way  to  compare  transaction  costs  across  different  programs. 


Following  the  method  used  by  Biggs  (2013),  this  study  uses  the  SEPM  cost 
ratio  for  a  program  as  shown  in  Equation  1: 


SE/PM  Cost  Ratio  = 


SE  +  PM  Costs 
Total  Cost 


(1) 


The  numerator  of  the  SE/PM  cost  ratio  is  the  sum  of  SE  and  PM  cost  expen¬ 
ditures  and  the  denominator  is  total  program  expenditures  (estimate  at 
completion,  or  E  AC).  A  ratio  is  calculated  to  provide  a  perspective  on  the  rel¬ 
ative  magnitude  of  SE/PM  expenditures  as  well  as  to  allow  for  comparison 
across  different  programs.  The  hypothesis  is  that  programs  with  higher  SE/ 
PM  cost  ratios  are  more  likely  to  experience  cost  breaches  than  programs 
with  lower  SE/PM  cost  ratios.  This  is  based  on  the  assumption  that  higher 
SE/PM  cost  ratios  are  related  to  riskier  contractual  relationships  since 
more  time,  effort,  and  resources  are  expended  to  meet  performance  and 
schedule  deadlines  when  compared  to  less  risky  contracts.  To  the  extent  that 
this  is  predicted  early  in  the  program,  it  could  be  useful  to  policy  makers  by 
providing  an  early  warning  that  programs  are  more  likely  to  result  in  cost 
and/or  schedule  overruns. 
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Biggs  (2013)  introduced  the  influence  diagram  in  Figure  1,  which  describes 
the  interactions  between  factors  that  may  be  associated  with  the  occurrence 
of  a  cost  breach.  The  dashed  lines  in  Figure  1  represent  factors  that  must 
be  dealt  with  qualitatively  or  by  using  proxies.  The  solid  lines  represent 
factors  that  can  be  quantitatively  evaluated.  While  the  risk  and  complexity 
of  a  program  may  directly  contribute  to  a  cost  overrun,  the  SE/PM  efforts 
and  the  contract  type  can  influence  the  magnitude  and  frequency  of  cost 
overruns  as  measured  by  cost  breaches. 


Source:  Biggs,  2013 


Figure  1  indicates  that  the  risk  and  complexity  of  the  MDAP  will  guide  pro¬ 
gram  managers  and  contractors  in  their  selection  of  an  appropriate  contract 
type,  which  in  turn  can  influence  the  government's  exposure  to  cost  over¬ 
runs.  In  all  likelihood,  the  risk  and  complexity  of  a  program  will  drive  the 
level  of  monitoring  and  negotiation  (transaction  costs)  required  to  manage 
the  program,  and  riskier,  more  complex  programs  will  require  higher  levels 
of  transaction  costs.  Specifically,  we  do  not  expect  that  transaction  costs 
themselves  drive  overruns,  but  rather  that  the  risk  and  complexity  that 
require  higher  levels  of  transaction  costs  drive  breaches. 
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Cost  Breaches 

In  this  article,  we  will  examine  how  the  SE/PM  ratio  and  contract  type 
are  related  to  the  probability  of  incurring  a  cost  breach.  Cost  breaches  occur 
when  the  amount  of  the  cost  overrun  exceeds  certain  parameters  defined 
by  regulation.  Within  the  defense  acquisition  community,  programs  may 
incur  two  types  of  cost  breaches:  Acquisition  Program  Baseline  (APB)  and 
Nunn-McCurdy  breaches.  For  a  program  to  incur  an  APB  breach,  estimated 
program  expenditures  must  be  greater  than  the  APB  EAC  by  at  least  10 
percent.  If  the  difference  is  15  percent  or  more,  a  Nunn-McCurdy  breach  is 
incurred.  Cost  breaches  frequently  are  incurred  in  six  categories  of  appro¬ 
priations:  average  procurement  unit  cost  (APUC);  program  acquisition  unit 
cost  (PAUC);  procurement;  research,  development,  test  and  evaluation 
(RDT&E);  military  construction  (MILCON);  and  acquisition-related  oper¬ 
ations  and  maintenance  (O&M).  Each  of  these  cost  breaches  was  included 
in  the  data  set  for  this  study. 


Within  the  defense  acquisition  community, 
programs  may  incur  two  types  of  cost  breaches: 
Acquisition  Program  Baseline  (APB)  and  Nunn- 
McCurdy  breaches. 

A  formal  cost  estimate  for  a  program  is  normally  developed  at  Milestone 
B,  the  initiation  of  an  acquisition  program,  when  the  APB  is  established. 
The  APB  is  used  for  tracking  and  reporting  cost  and  schedule  for  the  life 
of  the  program,  and  it  includes  threshold  and  objective  values  for  cost, 
schedule,  and  performance.  A  cost  threshold  breach  is  incurred  when  cost 
expenditures  exceed  the  cost  objective  by  10  percent  or  more  (DoD,  2013). 
If  an  MDAP  has  been  officially  rebaselined,1  cost  breaches  are  measured 
to  the  current  baseline. 

Nunn-McCurdy  cost  threshold  breaches  are  based  on  original  cost  esti¬ 
mates  for  PAUC  and  APUC  at  project  completion.  In  the  case  of  a  program 
that  has  rebaselined,  cost  threshold  breaches  are  also  based  on  the  current 
(i.e.,  rebaselined)  cost  estimate  for  PAUC  and  APUC  at  project  completion. 
For  purposes  of  this  study,  a  cost  breach  is  any  reported  in  the  SAR  that  is 
greater  than  or  equal  to  10  percent  above  the  APB.  The  type  of  cost  thresh¬ 
old  breach  and  the  APB  baseline  against  which  it  is  compared  are  shown 
in  Table  1. 
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TABLE  1.  APB  AND  NUNN-MCCURDY  COST  BREACH  THRESHOLDS 

APB  Breach 

Nunn-McCurdy 

Nunn-McCurdy 

(RDT&E, 

“Significant” 

“Critical” 

Procurement, 

Breach 

Breach 

MILCON,  O&M) 

(PAUC  &  APUC) 

Current  Baseline 
Estimate 

10% 

+15% 

+25% 

Original  Baseline 
Estimate 

N/A 

+30% 

+50% 

Source:  DoD,  2013 

Figure  2  can  be  used  to  illustrate  cost  overrun  calculations.  The  budgeted 
cost  of  work  performed  (BCWP)  represents  the  total  amount  budgeted  for 
work  packages  that  are  open  or  completed  at  any  given  point  in  time.  The 
budgeted  cost  of  work  scheduled  (BCWS)  represents  the  total  amount 
budgeted  for  the  work  that  was  scheduled  for  completion  at  a  given  point  in 
time.  The  actual  cost  of  work  performed  (AC  WP)  is  the  sum  of  actual  costs 
incurred  to  accomplish  the  work  performed  at  a  given  point  in  time. 


FIGURE  2.  EARNED  VALUE  MANAGEMENT 
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The  E  AC  is  the  sum  of  the  AC  WP  and  the  estimate  to 
completion  (ETC)  for  the  remaining  work.  The  ETC 
can  be  calculated  using  the  cost  performance  index 
(CPI)  and  the  schedule  performance  index  (SPI).  The 
formula  for  calculating  ETC  is: 

ETC  =  (BAC  -  BCWP )  /  ( CPI  *  SPI )  (2) 

When  the  EAC,  a  cost  estimate  for  the  total  cost  of 
the  contract,  is  higher  than  the  BAC,  the  baseline 
cost  estimate  of  the  contract,  a  cost  overrun  is  pro¬ 
jected.  Nunn-McCurdy  breaches  are  far  rarer,  perhaps 
because  of  the  political  threat  or  simply  the  higher 
threshold;  thus,  this  article  will  include  analysis  of  the 
potential  of  both  types  of  cost  overruns. 


Data 

This  study  used  data  originally  collected  by  Biggs 
(2013)  from  two  different  data  sources:  SARs  and  the 
Cost  and  Software  Data  Reporting  (CSDR)  System. 
The  SAR  contains  details  of  critical  parameters  of  an 
MDAP,  including  threshold  breaches,  schedule,  per¬ 
formance,  current  contracts,  and  cost  details.  MDAPs 
typically  require  several  contracts  to  be  executed, 
often  concurrently.  SARs  provide  information  for  the 
overall  program  and  not  for  individual  contracts.  A 
SAR  may  list  a  single  contract  or  many  contracts  for  a 
single  MDAP.  Because  threshold  breaches  are  associated  with  contract  esti¬ 
mates,  only  MDAPs  that  listed  one  contract  in  the  “Contracts”  section  of  the 
SAR  were  selected  for  purposes  of  this  study.  In  addition  to  cost  threshold 
breaches,  the  SAR  indicates  the  time  since  program  initiation  at  Milestone 
B,  which  was  used  in  this  study  to  indicate  program  maturity. 


The  program  cost  data  found  in  the  DD  Form  1921  CDSR  provided  by 
the  Defense  Cost  and  Resource  Center  in  the  Defense  Automated  Cost 
Information  Management  System  database  contain  significantly  more 
contract  detail  than  the  SARs.  The  WBS  format  of  the  CDSR  facilitates 
obtaining  information  on  SE/PM  costs.  To  simplify  the  data  collection 
process,  only  the  cost  data  provided  by  the  prime  contractor  were  recorded 
for  further  analysis.  The  SE/PM  cost  values  used  in  this  study  are  extracted 
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from  the  WBS  line  item  values  for  “EAC  SE/PM  cost,”  which  are  listed  on 
the  CDSR  (DD  Form  1921).  The  EAC  SE/PM  cost  is  the  projected  SE/PM 
cost  at  contract  completion.  The  SE/PM  costs  are  inclusive  of  the  total 
contract  costs  less  the  contractor's  profit/loss  or  fees. 

The  SE/PM  category  reported  by  the  contractor  has  some  limitations.  First, 
the  activities  included  in  this  category  will  vary  somewhat  from  contractor 
to  contractor.  As  a  result,  a  small  portion  of  the  differences  in  SE/PM  costs 
between  contracts  may  be  due  to  differences  between  the  accounting  sys¬ 
tems  used  by  each  contractor.  The  general  category,  however,  is  a  reasonable 
measure  of  the  cost  of  activities  commonly  associated  with  SE/PM.  Second, 
the  costs  included  in  SE/PM  for  a  single  contract  may  vary  over  time  as  new 
costs  are  defined  by  the  contractor  as  being  related  to  SE/PM.  This  could 
explain  a  small  portion  of  the  increase  in  the  SE/PM  cost  in  some  contracts, 
but  we  did  not  observe  significant  differences  in  our  data.  Finally,  we  rec¬ 
ognize  that  the  SE/PM  category  does  not  capture  all  possible  transaction 
costs  nor  was  it  ever  intended  to.  Rather,  it  is  likely  that  many  if  not  most 
of  the  activities  in  this  category  are  related  to  transactions,  as  opposed  to 
production,  and  thus  provide  a  reasonable  measure  of  transaction  costs. 

This  article  will  analyze  the  SE/PM-to-total-cost  ratios  of  MDAPs,  looking 
for  a  potential  correlation  between  these  ratios  and  the  probability  of  expe¬ 
riencing  a  cost  breach.  Determining  the  nature  of  any  potential  relationship 
between  the  SE/PM-to-cost  ratio  and  the  probability  of  cost  breaches  expe¬ 
rienced  by  a  program  will  test  the  hypothesis  that  programs  with  higher  SE/ 
PM  cost  ratios  will  experience  cost  overruns  more  frequently  than  programs 
with  lower  SE/PM  cost  ratios.  More  formally,  we  will  test  the  experimental 
hypothesis  Hx  that  the  probability  of  breach  is  increased  by  an  increase  in 
EAC  SE/PM  in  the  total  cost: 

Hx:  d(Probability  Cost  Breach)  /  d(EAC  SEPM/Total  cost)  >  0 

H0:  d(Probability  Cost  Breach)  /  d(EAC  SEPM/Total  cost)  =  0 

The  type  of  contract  used  for  the  program  was  also  obtained  from  the 
CDSRs.  Programs  were  noted  as  having  either  firm-fixed-price  type  con¬ 
tracts  or  cost-plus  type  contracts.  The  type  of  contract  used  for  a  program  is 
an  indication  of  the  perceived  level  of  risk  associated  with  execution  of  the 
contract.  As  the  level  of  performance  risk  increases,  the  risk  of  cost  overruns 
also  increases  and  the  amount  of  cost  risk  that  the  contractor  is  willing 
to  assume  tends  to  decrease.  Contract  types  differ  in  how  the  cost  risk  is 
shared  between  the  government  and  the  contractor.  In  a  firm-fixed-price 
contract,  no  cost  sharing  exists  between  the  government  and  the  contractor, 
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and  the  contractor  has  full  responsibility  for  the  performance  costs  and 
resulting  profit  (or  loss).  In  a  cost-plus  contract,  a  share  ratio  based  on  the 
contract  cost  and  the  contractor's  fee  (profit)  is  negotiated  so  that  the  con¬ 
tractor  has  a  predetermined  responsibility  for  the  performance  costs,  which 
will  directly  affect  the  fee  (profit)  (U.S.  General  Services  Administration, 
2005).  By  including  contract  type  in  our  analysis,  we  can  account  for  basic 
risk  differences  recognized  by  both  the  government  and  the  contractor  at 
the  outset  of  the  program.  Programs  with  aspects  of  both  were  treated  as 
cost-plus  type  contracts  since  cost-plus  contracts  are  abetter  indicator  of 
program  risk. 


TABLE  2.  DESCRIPTIVE  STATISTICS  1 

Variable 

Obs 

Mean 

Std.  Dev. 

Min 

Max 

Breach- 
Program  Av. 

32 

0.8125 

1.090649 

0 

4 

Nunn- 

McCurdy 

32 

0.28123 

0.5226715 

0 

2 

Program 

Type 

32 

0.516129 

0.5080005 

0 

1 

EAC  SEPM 
from 

Milestone  B 

32 

13.38844 

11.58719 

0.15 

42.85 

EAC  SEPM 

Program 

Average 

32 

14.66727 

10.68449 

0.87 

43.31 

To  Date 

SEPM 

Program 

Average 

30 

16.59583 

13.81136 

1.4 

54.66667 

Table  2  describes  the  data  used  in  our  study.  The  study  covers  32  programs 
over  84  program  years.  Despite  not  having  greatly  disparate  thresholds,  our 
sample  reflected  far  more  APB  breaches— 26  program  years— versus  only  9 
program  years  for  Nunn-McCurdy  breaches.  Approximately  half  of  the  pro¬ 
grams  were  firm-fixed-price  and  half  were  cost-plus  type  contracts.  We  also 
compared  three  different  measures  of  the  SE/PM  ratio.  The  first  measure  is 
for  the  EAC  of  the  SE/PM  ratio  to  total  costs,  as  estimated  at  Milestone  B. 
SE/PM  are  on  average  about  13  percent  of  total  costs  estimated  at  Milestone 
B.  Programs  that  rebaseline  would  update  this  measure,  but  for  purposes 
of  this  study  we  maintain  the  original  prebreach  measure  because  we  are 
interested  in  the  predictability  of  breaches  based  on  original  assessments  of 
risk.  The  second  measure  is  still  the  EAC,  but  programs  update  this  measure 
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as  they  go  along.  This  number  is  about  1.5  percentage  points  higher  than 
the  Milestone  B  estimate  or  14.7  percent  of  total  costs.  We  also  include  the 
actual  to  date  SE/PM  to  total  cost  at  each  point  in  time  for  our  sample.  At 
16.6  percent  of  total  costs,  this  is  even  higher  than  either  predicted  share, 
indicating  that  estimates  are  on  average  overoptimistic  and  that  SE/PM 
costs  grow,  on  average,  faster  than  total  costs. 

Analysis  of  the  data  shows  that  more  than  half  of  the  MDAPs  in  the  study 
initially  estimated  an  EAC  SE/PM  cost  ratio  of  0.10  or  less  and  experienced 
fewer  than  two  cost  breaches  since  1998.  Furthermore,  it  can  be  inferred 
that  most  of  the  programs  have  experienced  at  least  one  cost  breach,  which 
seems  to  confirm  a  RAND  report  finding  that  most  MDAPs'  actual  costs 
exceeded  baseline  cost  estimates  (Arena,  Leonard,  Murray,  &  Younossi, 
2006). 2  Observations  of  the  MDAP  SE/PM-to-total-cost  ratios  agree  with 
the  RAND  study,  suggesting  that  trends  in  SE/PM  costs  vary  across  MDAPs 
(Stem  et  al.,  2006).  Recall  that  SE/PM  costs  are  used  as  proxy  measures  of 
the  transaction  costs  required  to  administer  and  manage  the  MDAP. 


Methodology 

Because  our  data  set  includes  programs  that  have  not  experienced  cost 
breaches  over  the  time  period  studied,  our  data  are  considered  to  be  “right 
censored."  This  means  that  ordinary  linear  regression  is  not  a  good  option 
for  analyzing  the  data.  Instead,  we  employ  survival  analysis  to  test  whether 
relatively  high  Milestone  B  EAC  SE/PM  is  a  predictor  of  cost  breaches. 
Survival  analysis  is  typically  used  in  medicine  and  social  sciences  to  exam¬ 
ine  when  an  event  of  interest  will  occur.  For  example,  in  medicine  where  the 
event  of  interest  is  a  heart  attack,  we  can  use  survival  analysis  to  predict 
whether  a  patient  will  suffer  a  heart  attack  within  a  period  of  time.  In  this 
study,  the  event  of  interest  is  a  cost  breach,  and  we  are  interested  in  whether 
a  program  will  experience  a  cost  breach. 

In  our  medical  example,  we  could  use  survival  analysis  to  identify  risk  fac¬ 
tors,  such  as  obesity,  that  might  indicate  a  greater  propensity  for  suffering  a 
heart  attack.  In  this  study,  we  are  looking  for  risk  factors  that  might  predict 
cost  breaches.  Two  explanatory  variables  were  included  in  the  analysis: 
EAC  SE/PM  cost  ratio  and  program  contract  type  (fixed-price  or  cost-plus). 
While  the  exact  nature  of  the  relationship  between  cost  threshold  breaches 
and  these  explanatory  variables  is  unknown,  it  is  reasonable  to  presuppose 
that  the  explanatory  variables  influence  the  cost  performance  of  the  MDAPs 
as  shown  in  Figure  1. 
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Using  survival  analysis,  we  construct  a  hazard  function  for  cost  breaches. 
A  hazard  function  shows  (over  time)  the  probability  that  an  event  (such  as 
a  cost  breach)  will  be  incurred.  As  programs  can  experience  multiple 
cost  breaches  despite  rebaselining,  we  allow  for  multiple  breaches 
over  time3  and  estimate  how  the  hazard  of  cost  breach  varies 
with  our  explanatory  variables.  Hazard  models  are  also  useful 
because  they  are  more  tolerant  of  gaps  and  censoring.  Hazard 
models  can  be  thought  of  as  conditional  logits  (Cleves  et  al., 

2010).  We  allow  for  repeat  failures  over  the  period— that 
is,  following  cost  breaches,  we  allow  a  program  to  stay  in 
the  sample. 


Survival  analysis  uses  time-at-risk  as  its  relevant  time 
metric.  Thus,  we  measure  “survival  time”  in  terms  of  the 
maturity  of  the  design  and  technology  of  the  system.  In  this 
study,  program  maturity  is  measured  by  the  time  elapsed 
since  Milestone  B,  the  entry  point  into  the  Engineering  and 
Manufacturing  Development  phase.  For  a  program  to  receive 
approval  to  begin  Milestone  B  in  DoD,  the  design  and  technol¬ 
ogy  associated  with  the  system  must  be  considered  “mature.”4 

In  this  analysis,  we  use  the  Cox-Relative  Hazard.  It  is  con¬ 
sidered  semiparametric  because  it  does  not  imply  a  specific 
functional  form  on  the  hazard  of  breaches  over  time.  The 
proportional  hazard  model  is  specified  as: 

h.  (t)  =  h0  (t)  exp  {xfj) 

which  states  that  the  hazard  a  particular  subject./  faces  at 
time  t  is  a  function  of  the  baseline  hazard  modified  propor¬ 
tionally  by  the  vector  of  regression  coefficients  j3x.  The  Cox 
model  does  not  estimate  the  baseline  hazard.  We  can  convert 
coefficients  from  these  regressions  to  cumulative  hazard  ratios 
to  understand  the  marginal  effect  on  the  baseline  hazard  of  a  change 
in  the  coefficient.  This  is  done  simply  by  calculating  the  exponent  of  the 
coefficient  and  using  it  as  a  multiplier  (e.g.,  the  value  0.9  would  correspond 
with  a  10  percent  reduction,  and  1.1  would  be  a  10  percent  increase).  While 
we  report  coefficients,  we  interpret  our  results  using  exponentiated  hazard 
ratios— that  is,  the  cumulative  hazard  of  a  cost  breach. 
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We  next  examine  whether  baseline  SE/PM-to-cost  ratios  influence  the 
probability  of  a  Nunn-McCurdy  breach  using  a  similar  analysis.  While  the 
difference  between  the  two  is  mainly  the  10  percent  versus  15  percent 
threshold,  Nunn-McCurdy  breaches  are  sufficiently  rare  that  they 
may  have  significantly  different  causes. 


Finally,  we  test  the  robustness  of  our  finding  using  logit  mod¬ 
els.  Logit  is  commonly  used  to  determine  the  influence  of 
exogenous  variables  on  the  probability  of  a  dichotomous 
outcome,  such  as  whether  or  not  a  cost  breach  occurs  in  any 
given  program  year.  Logit  is  preferred  over  a  linear  regres¬ 
sion  model  because,  using  a  logistic  function,  it  constrains 
all  probability-of-occurrence  estimates  to  be  between  0 
and  1.  Formally,  the  logit  model  for  the  probability  can  be 
written  as: 

p(Breach)  =  I-r^ 


Where  xfl  is  a  function  of  the  vector  of  explanatory  variables. 

We  are  curious  as  to  whether  Milestone  B  SE/PM  predictions 
are  the  best  predictors  of  breaches  or  whether  within-program 
changes  in  SE/PM  estimates  or  actuals  should  cause  concern 
about  program  health.  To  test,  we  use  a  simple  logit  model, 
which  predicts  the  probability  of  failure  on  the  baseline 
EAC  SE/PM  ratios  as  in  our  hazard  model.  We  cluster  the 
standard  errors  by  program.  We  use  a  fixed  effects  (con¬ 
ditional)  logit  to  model  whether  a  change  in  either  EAC  or 
the  actual  SE/PM  cost  ratio  for  an  MDAP  will  change  its 
probability  of  breaching  a  cost  threshold.  Formally,  this 
model  measures  the  impact  of  deviations  by  the  independent 
variable  from  the  program's  mean  (Allison,  2001). 


Results 

We  find  that  higher  estimated  SE/PM  ratios  are  associated  with  a  higher 
risk  of  APB  breaches.  Figures  3  and  4  show  the  cumulative  risk  of  APB  and 
Nunn-McCurdy  breaches  over  program  maturity.  Table  3  shows  the  results 
for  Hazard  models  for  APB  breaches,  and  Table  4  shows  the  results  for 
Nunn-McCurdy  breaches 
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FIGURE  4.  HAZARD  OF  NUNN-MCCURDY  BREACH 
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TABLE  3. 

APB  BREACH  HAZARD 

Variables 

a) 

(2) 

EAC  SE/PM  from 

0.0482*** 

0.0284* 

Milestone  B 

(0.0147) 

(0.0168) 

Type 

1.125** 

(0.552) 

Observations 

84 

84 

Note.  Standard  errors  in  parentheses 

***  p<0.01,  **  p<0.05,  *  p<0.1 

In  the  first  model  a  1  percentage  point  increase  in  the  estimated  SE/PM 
ratio  at  completion  increases  the  risk  of  breach  by  5  percent,  a  result  that  is 
statistically  significant  at  the  1  percent  level.  When  contract  type  is  added 
as  a  control,  the  impact  of  the  SE/PM  ratio  goes  down  to  3  percent  and  its 
significance  is  reduced  to  the  10  percent  level.  Looking  at  the  impact  of 
contract  type,  we  find  that  having  a  cost-plus  type  contract  multiplies  the 
risk  of  an  APB  breach  by  3.1,  which  is  significant  at  the  5  percent  level. 


TABLE  4.  NUNN- 

MCCURDY  BREACH  HAZARD 

Variables 

(3) 

(4) 

EAC  SE/PM  from 

0.0247 

0.00352 

Milestone  B 

(0.0268) 

(0.0290) 

Type 

1.269 

(0.886) 

Observations 

84 

84 

Note.  Standard  errors  in  parentheses 

***  p<0.01,  **  p<0.05,  *  p<0.1 

Nunn-McCurdy  breaches  are  significantly  less  common  than  APB  breaches 
and  SE/PM  ratios  or  contract  type  do  not  appear  to  determine  them.  The 
SE/PM  ratio  is  not  a  significant  predictor  of  Nunn-McCurdy  breaches,  and 
while  type  has  a  similar  magnitude  to  its  impact  on  APB  breaches,  it  is  not 
statistically  significant  either. 
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TABLE  5.  LOGIT  ANALYSIS 

Variables 

<i) 

Logit 

(2) 

Fixed  Effects 

(3) 

Fixed  Effects 

Baseline  EAC  SE/PM 

0.0433* 

(0.0244) 

To  Date  SE/PM 

-0.0358 

(0.0491) 

EAC  SE/PM 

-0.0924 

(0.0782) 

Constant 

-1.433*** 

(0.508) 

Observations 

84 

36 

39 

Number  of  programs 

10 

11 

Table  5  shows  the  results  of  logit  analysis  for  APB  breaches.  We  find  that 
the  likelihood  of  a  cost  breach  is  related  to  the  EAC  SE/PM  ratio  predicted 
at  milestone  B,  but  not  to  changes  in  the  predicted  ratio  or  to  the  actual  SE/ 
PM  ratio.  The  marginal  effect  of  the  logit  model  is  very  similar  to  the  hazard 
function  and  statistically  significant  at  the  10  percent  level:  for  every  1  per¬ 
cent  increase  or  decrease  in  the  EAC  SE/PM  cost  ratio,  there  is  an  increase 
or  decrease  of  4  percent  in  the  probability  of  a  program  sustaining  a  cost 
threshold  breach.  We  do  not  find  any  within-program  impact  using  fixed 
effects  logit  models,  indicating  that  a  change  in  the  EAC  SE/PM  ratio  is  not 
an  indicator  of  a  potential  breach.  Thus  forecasts  of  cost  breaches  should 
focus  on  initial  SE/PM  ratio  estimates. 


Conclusion 

This  article  successfully  tested  the  hypothesis  that  transaction  costs  could 
help  explain  future  cost  breaches  of  MDAPs.  Using  SE/PM  costs  as  a  ratio  of 
total  program  costs,  we  find  the  greater  this  ratio  is  at  the  outset  (Milestone 
B  estimate),  the  greater  the  risk  of  eventual  cost  breaches.  This  information 
reflects  the  program  manager's  implicit  assessment  of  the  risk  of  a  program 
and  can  be  a  valuable  early  indicator  of  which  programs  will  benefit  from 
greater  oversight.  We  should  be  careful  to  note  that  high  SE/PM  ratios  may 
be  the  result  of  program  managers  responding  proactively  to  program  risks. 
As  such,  we  are  not  suggesting  that  high  SE/PM  ratios  are  a  bad  thing— simply 
that  they  can  be  used  to  predict  cost  breaches,  which  often  occur  in  high-risk, 
more  complex  programs.  This  suggests  that  the  SE/PM  ratio  is  a  measurable 
indicator  of  cost  risk  and  anticipated  transaction  costs. 
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This  article  also  offers  an  innovative  way  of  looking  at  cost  breaches,  using  sur¬ 
vival  analysis  to  forecast  cost  breaches  in  MDAPs  over  a  program's  life  cycle. 
We  offer  somewhat  robust  analysis,  using  a  more  traditional  logit  analysis,  of 
our  predictions.  We  believe  this  is  a  step  forward  in  using  measures  available 
in  DoD's  vast  databases  of  cost  information  to  develop  more  robust  forecasts 
of  potential  cost  overrun  risks  in  programs.  These  findings  suggest  that  the 
department  could  benefit  from  capturing  more  explicit  measures  of  transac¬ 
tion  costs  to  determine  more  precisely  their  role  in  predicting  cost  variability. 


Appendix 

PROGRAMS  SELECTED  FOR  STUDY 

Active  Electronically  Scanned  Array  (AESA)  Radar 
AIM-9X/Short  Range  Air-to-Air  Missile 

AIM-120  Advanced  Medium  Range  Air-to-Air  Missile  (AMRAAM) 

Airborne  and  Maritime/Fixed  Station  Joint  Tactical  Radio  System  (AMF  JTRS) 
AN/WQR-3,  Advanced  Deployable  System  (ADS) 

Apache  Block  IMA  Remanufacture  (AB3A  REMANUFACTURE) 

AV-8B/Attack,  V/STOL,  Close  Air  Support  (Harrier  11+  Remanufacture) 

B-2  Radar  Modernization  Program 

Cobra  Judy  Replacement  (Cobra  Judy  Replacement) 

EA-18G  Growler  (EA-18G) 

Expeditionary  Fighting  Vehicle  (EFV) 

E-3  AWACS  Radar  System  Improvement  Program  (RSIP) 

E-2C  Reproduction 

Family  of  Advanced  Beyond  Line-of-Sight  Terminals  (FAB-T) 

Family  of  Medium  Tactical  Vehicles  (FMTV) 

Guided  Multiple  Launch  Rocket  System/DPICM/Unitary/Alternative  Warhead 
(GMLRS/GMLRS  AW) 

Joint  Common  Missile  (JCM) 

Joint  Tactical  Radio  System  Ground  Mobile  Radio  (formerly  Cluster  1)  (JTRS  GMR) 
Longbow  Hellfire  -  subsystem  of  the  AH-64  Apache  Weapon  System 
LHA  Replacement  Amphibious  Assault  Ship 

MQ-4C  Unmanned  Aircraft  System  Broad  Area  Maritime  Surveillance  (MQ-4C 
UAS  BAMS) 

Multi-Platform  Radar  Technology  Insertion  Program  (MP-RTIP) 

National  Polar-orbiting  Operational  Environmental  Satellite  System  (NPOESS) 
Presidential  Helicopter  Replacement  (VH-71)  Program 
P-8A  Poseidon 

Sense  and  Destroy  Armor  (SADARM) 

Small  Diameter  Bomb  Increment  II  (SDB  II) 

Space  Based  Infrared  System  (SBIRS)  High  Program 
Standard  Missile  (SM)  -  2  Block  IV 
Stryker  Family  of  Vehicles  (STRYKER) 

UH-72A  Light  Utility  Helicopter  (LUH) 

Warfighter  Information  Network  -  Tactical  (WIN-T) 
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Endnotes 

Circumstances  authorizing  changes  to  the  baseline  are  limited;  revisions  to  the 
current  APB  are  not  authorized  unless  there  is  a  significant  change  in  program 
parameters  and  must  be  approved  by  the  Milestone  Decision  Authority  (DoD,  2013). 

2  For  most  of  the  programs  reviewed,  actual  costs  exceeded  the  baseline  cost 
estimate  established  at  Milestone  B  (program  initiation),  as  measured  by  the  cost 
growth  factor  (Arena  et  al.,  2006). 

3  For  robustness,  we  also  ran  the  analysis  without  allowing  for  multiple  breaches,  and 
while  the  results  were  weaker,  they  were  still  statistically  significant  and  of  a  similar 
magnitude.  Still,  the  fact  that  including  multiple  breaches  strengthens  our  results, 
indicates  that  programs  with  high  levels  of  complexity  and  risk  often  suffer  for  these 
high  levels  repeatedly. 

4 Milestone  B  approval  authorizes  an  MDAP  to  enter  the  Engineering  and 
Manufacturing  Development  phase  of  the  acquisition  process.  Statutory  requirements 
for  MDAPs  to  achieve  Milestone  B  approval  are  found  in  Title  10  U.S.C.  §  2366b. 

These  requirements  stipulate  that  the  program  be  certified  by  the  Milestone  Decision 
Authority  to  be  affordable,  fully  funded  through  the  Future  Years  Defense  Program 
(FYDP),  and  that  the  cost  and  schedule  estimates  are  reasonable. 
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